Vision based calibration system for unmanned aerial vehicles

ABSTRACT

A system includes a camera configured such that a field of view of the camera is positioned to capture a base portion and a movable portion of an aircraft. A computer communicatively connected to the camera is configured to determine a relative orientation between a base identification feature of the base portion and a movable identification feature of the movable portion of the aircraft based on image data received from the camera. The computer is also configured to determine a position of the movable portion of the aircraft based on the relative orientation between the base identification feature and the movable identification feature and a position of the base portion and to compare the position of the movable portion to a reported position for the movable portion to determine a variance. An aircraft control system is configured to monitor a state of the aircraft based on the variance.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation patent application of U.S.Non-provisional application Ser. No. 16/017,831, filed Jun. 25, 2018,and titled “Vision Based Calibration System for Unmanned AerialVehicles”, issued as U.S. Pat. No. 10,793,269 on Oct. 6, 2020, which isa continuation patent application of U.S. Non-provisional applicationSer. No. 15/229,099, filed Aug. 4, 2016, issued as U.S. Pat. No.10,029,788 on Jul. 24, 2018, and titled “Vision Based Calibration Systemfor Unmanned Aerial Vehicles,” which claims the benefit to U.S.Provisional Patent Application No. 62/314,323, filed Mar. 28, 2016, andtitled “Unmanned Aircraft System for High Volume Operations,” thedisclosures of each of which are hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

This disclosure relates to a vision based calibration system forUnmanned Aerial Vehicles (UAVs) that is designed for reliable,high-volume operations.

BACKGROUND

Autonomous aerial vehicles have been gaining popularity in the last fewyears. UAVs have gone from a rarity to a common sight. UAVs are used notonly by hobbyists and recreational flyers, but also by governments andbusinesses, for purposes such as surveillance, mapping, and mostrecently, aerial delivery.

In order to conduct high volume operations economically with a UnmannedAircraft System (UAS), the UAVs and ground equipment must be designed ina way that allows for the aircraft to be easily tested, calibrated,launched, and recovered.

SUMMARY

Embodiments discussed herein are related to methods and systems forvision based calibration and testing of UAVs. In one embodiment a camerais configured to be positioned relative to a vehicle (e.g. a UAV) suchthat a base fiducial and a movable fiducial on the vehicle are visiblewithin a field of view of the camera. A testing and calibration computeris then used to receive an image of the base fiducial and the movablefiducial from the camera. The image comprises image data. The computerthen determines a relative orientation between the base fiducial and themovable fiducial based on the image data.

The relative orientation between the base fiducial and movable fiducial,and position information about a base surface of the vehicle, is used todetermine a real position of a movable part of the vehicle. The realposition of the movable part is compared to a reported position for themovable part and a notification is generated if the difference betweenthe real position and reported position is greater than a thresholdamount.

A system described herein includes a camera configured such that a fieldof view of the camera is positioned to capture a base portion and amovable portion of an aircraft. A computer communicatively connected tothe camera is configured to determine a relative orientation between abase identification feature of the base portion and a movableidentification feature of the movable portion of the aircraft based onimage data received from the camera. The computer is also configured todetermine a position of the movable portion of the aircraft based on therelative orientation between the base identification feature and themovable identification feature and a position of the base portion and tocompare the position of the movable portion to a reported position forthe movable portion to determine a variance. An aircraft control systemis configured to monitor a state of the aircraft based on the variance.

A method described herein captures, with a camera, an image of a firstportion of an aircraft including a first identification feature and asecond portion of the aircraft including a second identificationfeature, where the image includes image data. A computer communicativelyconnected to the camera determines, using the image data, a position ofthe second portion of the aircraft based on a relative orientationbetween the first identification feature and the second identificationfeature. A state of the aircraft is updated based at least on theposition of the second portion of the aircraft.

A method for predicting failures in a movable portion of an aircraftincludes a data collection process, a model development process, and adiagnostic process. The data collection process includes capturing, witha camera, an image of a base identification feature of a base portion ofthe aircraft adjacent to the movable portion and a movableidentification feature of the movable portion, where the image includesimage data. A relative orientation between the base identificationfeature and the movable identification feature is determined based onthe image data. A real position of the movable portion of the aircraftis determined based on the relative orientation between theidentification features and a known real position of the base portion ofthe aircraft. An actuator variance is generated based on a differencebetween the real position of the movable portion and a reported positionfor the movable portion. The model development process includesexecuting the data collection process a plurality of times over a periodof time for the movable portion of the aircraft and storing a pluralityof actuator variances generated by the data collection process over theperiod of time, observing a failure in the movable portion, anddeveloping a predictive model based on the stored plurality of actuatorvariances. The diagnostic process includes executing the data collectionprocess a second plurality of times for a second movable portion of asecond aircraft over a second period of time and storing a secondplurality of actuator variances generated by the data collection processover the second period of time. The predictive model is applied to thesecond plurality of actuator variances and a notification is generatedbased on output of the predictive model.

BRIEF DESCRIPTION OF THE DRAWINGS

The figures use like reference numerals to identify like elements. Aletter after a reference numeral, such as “100 a,” indicates that thetext refers specifically to the element having that particular referencenumeral. A reference numeral in the text without a following letter,such as “100,” refers to any or all of the elements in the figuresbearing that reference numeral (e.g. “100” in the text refers toreference numerals “100 a” and/or “100 b” in the figures).

FIG. 1A is a diagram illustrating the components of a UAS and entitiesthat may interface with it, according to one example embodiment.

FIG. 1B is a diagram illustrating a UAV launch process, according to oneexample embodiment.

FIG. 2A is a diagram illustrating a top perspective view of an exampledesign of a UAV, according to one embodiment.

FIG. 2B is a diagram illustrating a bottom perspective view of anexample design of a UAV, according to one embodiment.

FIG. 2C is a diagram illustrating a bottom perspective view of anexample design of a UAV with tail hook extension boom, according to oneembodiment.

FIG. 2D is a system diagram illustrating the components of a UAV,according to one example embodiment.

FIG. 3 is a diagram illustrating the components of a distributioncenter, according to one example embodiment.

FIG. 4 is a diagram illustrating the components of the global services,according to one example embodiment.

FIG. 5A is a diagram illustrating an example testing and calibrationsystem, in a top view according to one embodiment.

FIG. 5B is a diagram illustrating an example testing and calibrationsystem, in a top view, with a UAV placed upon it, according to oneembodiment.

FIG. 5C is a diagram illustrating an example testing and calibrationsystem, with a UAV placed upon it, in a side view, according to oneembodiment.

FIG. 5D is a diagram illustrating an example testing and calibrationsystem, showing a camera, base fiducial, and movable fiducial, accordingto one embodiment.

FIG. 5E is a diagram illustrating a close up view of an example camera,according to one embodiment.

FIG. 5F is a diagram illustrating an example process for testing andcalibration using a camera-based system, according to one embodiment.

DETAILED DESCRIPTION

While the present disclosure has been described with reference tovarious embodiments, it will be understood that these embodiments areillustrative and that the scope of the disclosure is not limited tothem. Many variations, modifications, additions, and improvements arepossible. Modules and procedures may be separated or combined indifferent ways in various embodiments, or described with differentterminology. These and other variations, modifications, additions, andimprovements may fall within the scope of the disclosure as defined inthe claims that follow.

This document describes a testing and calibration system designed foruse with a high volume UAS operation. The UAS includes a UAV designed tooperate substantially independently of a ground operator, a testing andcalibration system, a UAV launcher, and a recovery system. The systemalso includes software components, some that may execute on the UAV,some that may execute on computer servers running locally, and some thatmay execute on remote cloud servers.

The UAS described herein provides a platform for performing varioustarget functions, including, but not limited to, package delivery, datacapture, mapping, surveillance, and infrastructure provisioning. Whilespecific embodiments of the UAS system are described herein, someembodiments may comprise systems and methods that are not generallyrelevant to every target function. One skilled in the art will readilyrecognize the relevance of a recited system or method in relation to thetarget functions.

Unmanned Aerial System (UAS)

The UAS disclosed in this document is an example implementation of anunmanned aircraft system designed for safe, reliable, high-volumeoperations. In this embodiment the aircraft are fixed-wing UAVs, but theUAS may also be used in conjunction with other aircraft designs, such asquad copters, lighter-than-air vehicles (such as blimps), gliders,tilt-rotor aircraft, etc. Moreover, as used herein, the term“autonomous” may refer to operations of an unmanned vehicle that areperformed by the vehicle without user intervention and/or control, aswell as to describe vehicles that are designed to operate without humanintervention and/or control for all or portions of their missions.Accordingly, a vehicle and/or a system may be described as autonomouseven though a human operator may choose to override the vehicle'sautonomous control.

FIG. 1A illustrates an embodiment of a UAS and interfacing entities. Inthis embodiment, the UAS 100 receives a service request from a servicerequestor 104 and deploys a UAV 102 to fulfill that request. In theevent that the UAV 102 encounters a situation that its onboardautomation cannot understand or handle (or a human operator becomesaware of a situation that warrants human intervention), the UAS 100 isable to provide human intervention by selecting a remote vehicleoperator 108 who can issue commands to the UAV 102. In this embodiment,the UAS 100 comprises a distribution center 101, a UAV 102, and globalservices 103.

The service requestor 104 is a human user or an autonomous system thatissues a service request to the UAS 100. In the case where the servicerequestor 104 is a human user, that user may use a remote client devicesuch as a mobile phone, tablet, or personal computer to issue therequest. A service request is an instruction to the UAS 100 to providesome service at the destination site 105. The destination site 105 maybe any designated location, such as a portion of open ground, abuilding, a mailing address, a Global Positioning System (GPS)coordinate, or a slice of airspace. In some embodiments, the destinationsite 105 is the location of a beacon device. The beacon device may beany device that emits a signal that can be used to track or identify alocation, such as for example a transponder, a mobile phone, etc. Thedestination site 105 may also be designated by identifying a particularobject, such as, for example, a designated vehicle, a mailbox, adelivery pad, or some other target object that can be tracked toindicate a target location for a service. In another embodiment, thedestination site 105 is the location of the service requestor 104,although this need not be the case. Although one service requestor 104and one destination site 105 are illustrated in this embodiment, inpractice there can be many service requestors 104 and destination sites105.

The requested service may be any service that can be provided from anairborne platform. For example, in one embodiment, the service requestissued by the service requestor 104 is a request to deliver a packagecontaining a specific payload to the destination site 105. In anotherembodiment, the service request is a request to capture image data usinga camera mounted on the UAV 102, at the destination site 105 or along aroute to and from the destination site 105. In yet another embodiment,the service request is a request to provide an Internet access point atthe destination site 105 using a Wi-Fi gateway mounted on the UAV 102.Many other services can be provided using the UAS 100 at the destinationsite 105, such as package pickup, surveillance, mapping, data captureusing UAV-mounted instruments, etc.

The UAV 102 is an Unmanned Aerial Vehicle. The UAV 102 can beimplemented using a variety of systems and airframes. Almost anypractical flying platform can be used as the vehicle base for the UAV102, including gliders, airplanes, balloons, helicopters, etc. In oneembodiment, the UAV 102 is implemented using a fixed-wing aircraft withredundant propulsion systems that is optimized for long-range flight. Inanother embodiment, the UAV 102 is implemented using a quad-rotoraircraft that is optimized for short-range flight and vertical takeoffand landing. In yet another embodiment, the UAV 102 is implemented usinga hybrid fixed-wing aircraft, with tilt-rotors, capable of bothlong-range flight and vertical takeoff and landing. In anotherembodiment, the UAV 102 is implemented using a fixed-wing aircraft withfixed horizontally oriented motors, configured to provide horizontalthrust, and separate fixed vertically oriented motors configured toprovide vertical thrust. The UAV 102 may also be implemented using alighter than-air-platform such as a balloon, blimp, or other dirigible.One purpose of the UAV 102 in the UAS 100 is to serve as a flexibleplatform that can be rapidly deployed on demand, with minimum humaninvolvement.

Although the UAV 102 is an autonomous vehicle that is designed tooperate without human assistance in most scenarios, it may occasionallyrequire the intervention of a human controller or pilot. For instance, aglobal systems operator 106 or a distribution center operator 107 mayissue a recall command to the UAV 102 while it is on a mission, due tosome external issue, such as inclement weather, a canceled delivery,etc. The UAV 102 may also proactively request human assistance while itis on its mission. For example, the UAV 102 may encounter an environmentwhere its vision and/or navigation algorithms cannot produce a path witha high degree of reliability. In such a scenario, the UAV 102 will senda request for assistance to the global services 103. The global services103 will select a remote vehicle operator 108 to handle the situation,and that operator can send the UAV 102 one or more commands to help itnavigate its environment.

The UAV 102 may carry any suitable payloads, depending on the nature ofthe service request received from the service requestor 104. Componentsof the UAV 102 are explained in more detail in the description for FIGS.2A-2D. Although a single UAV 102 is depicted in FIG. 1A, there may bemore than one UAV 102 in a UAS 100.

The distribution center 101 is a fixed or mobile facility thatfacilitates the launch, recharge, communications, repair, and payloadlogistics for the UAV 102. The distribution center 101 is explained infurther detail in the description for FIG. 3 . Although a singledistribution center 101 is shown in FIG. 1A, there may be more than onedistribution center 101 in the UAS 100. In one embodiment, each UAV 102in the UAS 100 is based at a single distribution center 101, and isrepaired, reloaded, and recharged at that distribution center 101. Inanother embodiment, each UAV 102 can be repaired, reloaded, andrecharged at any distribution center 101 in the UAS 100, and UAVs 102may be routed between distribution centers 101 based on the logisticalrequirements of current service requests and the projected requirementsfor future service requests. Each distribution center 101 may have alauncher system and a recovery system that are capable of automated,reliable, high-volume launches and recoveries of UAVs 102. Eachdistribution center 101 may also include equipment for testing andcalibration of UAVs 102 prior to launch and/or after recovery.

The global services 103 may be comprised of one or more computer serversystems, running software services (i.e. computer software programs),accessible through the Internet, which provide offsite support,administration, air traffic control, communications, data storage andlogistics functions for the distribution centers 101 and the UAVs 102.In one embodiment, the global services 103 route a service request froma service requestor 104 to a distribution center 101 that isgeographically adjacent to (or in relative geographic proximity to) thedestination site 105.

The global services 103 may also receive requests for assistance fromthe UAV 102 while it is on its mission. Based on such requests, theglobal services 103 will select a remote vehicle operator 108 from apool of operators, and provide data about the UAV 102's environment tothe remote vehicle operator 108. Based on this provided data, the remotevehicle operator 108 can provide one or more commands to the UAV 102 tohelp it surmount any problems that its on-board intelligence cannothandle. The global services 103 are explained in more detail in thedescription for FIG. 4 .

The global system operator 106 may be a human user that monitors andoperates the UAS 100 to ensure the correct and efficient functioning ofthe system. For example, in some embodiments, the global system operator106 may monitor the UAS 100 through the computer servers of the globalservices 103, to ensure that a distribution center 101 has theappropriate payload in stock to fulfill a service request from a servicerequestor 104. In one example embodiment, the global system operator 106may use the global services 103 to route new stock of a particularpayload to a distribution center 101 in anticipation of that payloadstock being depleted.

There may be more than one global system operator 106, and the globalsystem operators 106 may monitor and provide services for multipledistribution centers 101, UAVs 102, and service requestors 104.

The distribution center operator 107 is a human user that monitors andoperates the distribution center 101. The distribution center operator107 may ensure that the UAS 100 components that are local to thedistribution center 101 function correctly. This includes the UAVs 102based at the distribution center 101, as well as other components suchas launchers, recovery systems, testing and calibration systems,rechargers, payloads, etc. The distribution center 101 provides systemsand methods to facilitate the tasks of the distribution center operator107. For example, in some embodiments, the distribution center operator107 operating a distribution center 101 is provided with an operatorinterface that allows her to determine the inventory of each type ofpayload at that distribution center 101, and that enables her to ordermore of any type of payload that is in short supply. The distributioncenter operator 107 may also operate the launcher system, recoverysystem, and testing/calibration systems located at that distributioncenter 101 using the operator interface. The distribution center systemsand methods that facilitate the distribution center operator 107's workare explained in more detail in the description for FIG. 3 .

The remote vehicle operator 108 is a human user that receivesinformation about the UAV 102 from the global services 103 and may issuecommands to the UAV 102 to help it complete its mission. In oneembodiment of the system there is a pool of available remote vehicleoperators 108 that can provide assistance to any UAV 102 in the system.When the global services 103 receive a request for assistance from a UAV102, it selects from among the available remote vehicle operators 108and routes the request to that operator. The remote vehicle operator 108reviews information about the circumstances of the UAV 102 and sends oneor more commands to the UAV 102. Based on these commands, the UAV 102takes actions that help it to complete its mission. In one embodiment,the roles of the global system operators 106 and the remote vehicleoperators 108 are merged.

FIG. 1B illustrates one embodiment of a UAV launch process implementedby the UAS 100. As an initial step, the global services 103 of the UAS100 receive 150 a service request from a service requestor 104. Theservice request specifies a destination site 105, which designates thelocation where the service is to be delivered. As described herein, theservice request may also include payload information, corresponding to apayload requested by the service requestor. The global services 103 thenselect 151 a suitable distribution center 101 from which to fulfill theservice request. In some embodiments, the global services 103 select 151the distribution center 101 from which to fulfill the service request bydetermining the distribution center 101 that is closest to the locationof the destination site 105. In another embodiment, the global services103 select 151 a distribution center 101 to fulfill the service requestby taking into account both the proximity of the distribution center 101to the destination site 105 as well as an inventory at the distributioncenter 101 that indicates the availability of a payload specified in theservice request. For example, if the service request is a request todeliver a specific type of item to the destination site 105, the globalservices 103 will select the distribution center 101 from thosedistribution centers that are near the destination site 105 and thathave the requested item in their inventory. Other factors can also beused to select a distribution center 101, such as, for example, thelocal weather conditions and air traffic at the distribution centers101.

Once a distribution center 101 is selected 151, at least a portion ofthe information in the service request is sent 152 to that distributioncenter 101. In addition to the destination site location and payloadinformation, the service request may contain other information that isuseful at the distribution center 101 for the fulfillment of the servicerequest. For example, in some embodiments, the service request furthercomprises a time designating when the service request should befulfilled at the destination site 105.

A UAV 102 can be selected 153 to fly a mission to fulfill the request,either during the distribution center selection process or afterwards.The UAV 102 that will fly the mission may be selected 153 based on oneor more criteria that are relevant to the service request and/or systemefficiency. For example, in one embodiment, the UAV 102 is selected 153based on the charge level of its battery and the distance to thedestination site 105. In another embodiment, the UAV 102 is selected 153based on the instruments that are installed on its airframe and a typeof data capture specified in the service request. In yet anotherembodiment, the UAV 102 is selected 153 based on a package in itspayload matching a package specified for delivery in the servicerequest.

In an alternative embodiment, the UAS 100 does not select frompre-configured UAVs for a given mission. Instead, either thedistribution center 101 or the global services 103 determine a set ofcomponents that are required to complete the service request, and thedistribution center 101 causes a UAV comprising the required componentsto be assembled for the mission. For example, if the destination site105 is a certain distance from the distribution center 101, the UAV forthe mission can be configured with a suitable battery pack and enginesto complete a round-trip flight to that destination.

The selection 153 of the UAV 102 may occur after the selection 151 ofthe distribution center, or may be used as a factor in selecting 151 thedistribution center 101. For example, the distribution center 101 may beselected 151 from only those distribution centers that have a particulartype of UAV airframe, UAV battery, or UAV engine, based on the weight ofa payload required by the service request.

Once the UAV 102 is selected 153 for the mission, mission data isgenerated 154 for it. The mission data is information that enables theUAV 102 to navigate to the destination site 105 and fulfill the servicerequest. In some embodiments, the mission data includes GPS coordinatesfor the destination site 105 as well as flight corridor informationfacilitating navigation to those GPS coordinates. The flight corridorinformation is discussed in more detail in the descriptions for FIG. 2Dand FIG. 3 . Further details related to the mission data are discussedin the descriptions for FIG. 2D, FIG. 3 , and FIG. 4 . After the missiondata is generated 154, it is uploaded into a database on the UAV 102.

Once the mission data is generated and uploaded 154, the UAV 102 islaunched 155. From the time the UAV 102 is launched and until it landsagain, it is considered to be on a mission to complete the servicerequest. In one embodiment, the UAV 102 may be launched with a missionto fulfill more than a single service request. In another embodiment, atleast a part of the mission data is uploaded and perhaps even generated,after the UAV 102 is launched 155.

Unmanned Aerial Vehicle (UAV)

The UAV 102 is an aircraft system with hardware and software modulesthat enable it to fulfill service requests with little or no humansupervision. In one embodiment, the UAV 102 is comprised of an airframethat includes hardware and software modules that enable it to flyautonomously and complete a service request. FIG. 2A illustrates anupper perspective view of an example embodiment of a UAV 102. Theembodiment illustrated comprises wings 211, tail 212, and controlsurfaces 213. The control surfaces 213 are coupled to and moved byactuators (not depicted in the figure). In one embodiment, the actuatorsare electrical servo motors. In another embodiment the actuators arepneumatic. The UAV 102 also has one or more engines, which are used forpropulsion. The engines are not depicted in the illustration depicted inFIG. 2A, but they may be mounted on the wings 211, the nose of theaircraft, or the fuselage. The number and position of the engines can bevaried based on the requirements of the aircraft. The engines may be jetengines, electric power propeller engines, gasoline-powered propellerengines, or any other type of engine that is suitable for use in lightaircraft.

In the illustrated embodiment the UAV is a fixed-wing aircraftconstructed of carbon fiber and plastic, with a V-tail comprising dualruddervators. Alternative airframe configurations and materials may alsobe used for the UAV 102. For example, the UAV 102 may be a quadcopter, afixed-wing aircraft with a conventional tail, a hybrid aircraft withboth wings and vertical take-off rotors, etc. The materials used for theaircraft may be carbon fiber, titanium and aluminum alloys, plastic, orany other durable, lightweight material.

FIG. 2B illustrates a lower perspective view of the same embodiment ofthe UAV 102. This view of the aircraft shows the calibration fiducials214, the payload bay doors 215, and the tail hook 216. The calibrationfiducials 214 are geometric patterns that enable a vision-based testingand calibration system to determine the position and movement of acontrol surface 213. The calibration fiducials and the testing andcalibration system are used during pre-flight checks to ensure that theUAV 102's control surfaces 213 move in the expected manner when servosfor those control surfaces are activated.

The payload bay doors 215 give access to an interior payload bay withinthe UAV 102. The payload bay can be used to carry packages for delivery,instruments for surveillance missions, communications equipment, etc.The payload bay doors 215 can be opened while the UAV 102 is in flightin order to airdrop a package at a destination site 105. The interiorpayload bay may include a mechanism that is used to expel a carriedpackage from the payload bay. The mechanism used to expel the packagefrom the payload bay may be a compressed spring, an elastic band, aservo motor, a hydraulic mechanism or any other device that can be usedto forcefully expel the payload so that it can reliably clear theaircraft during a delivery.

In one embodiment, the UAV 102 carries its payload externally, attacheddirectly to the aircraft fuselage, instead of in an internal payloadbay. In the case of an externally carried payload, the aircraft fuselagewill have attachment points for a payload, which may be actuated inorder to release the payload.

The tail hook 216 is a durable hook-like structure attached to the tailor fuselage of the UAV 102. The tail hook 216 assists in the recovery ofthe UAV 102. In the illustrated embodiment, the tail hook 216 isattached directly to the tail 212 of the UAV 102, but in an alternativeembodiment, as shown in FIG. 2C, the tail hook 216 may be attached atthe end of an extension boom 217, which extends beyond the end of thetail 212. The extension boom may be coupled to a boom hinge 218 whichallows the tail hook 216 to be pivoted away from the tail 212, to allowfor a separation between the main body of the UAV 102 and the tail hook216. This separation can assist in the safe recovery of the UAV 102.

FIG. 2D is a block diagram of the systems of a UAV 102 according to oneexample embodiment. The embodiment of the UAV 102 illustrated in FIG. 2Dcomprises a mission planner 200, a flight controller 201, a sensorsystem 202, a communications system 203, an actuator control system 204,a propulsion management system 205, a payload management system 206, anda safety system 207. Collectively, these modules together comprise anaircraft control system 210 for the UAV 102. In an embodiment of the UAV102, two or more of the modules mentioned above may be combined into asingle hardware component to reduce complexity, improve reliability,reduce weight, and/or reduce cost. For instance, in one exampleembodiment, the mission planner 200 and the flight controller 201 may beimplemented using software modules that run on the same System On Chip(SOC) hardware.

Although not depicted in the figure, the modules of the UAV 102 areinterconnected via at least one communications bus. The bus allows themodules to communicate with each other to receive and send informationand commands. The bus may be implemented using any of the methods knownto those with familiarity in aviation and vehicle engineering. Forexample, the bus may be implemented using the Controller Area Network(CAN) standard. To improve the reliability of the system, embodimentsmay use additional redundant buses. For example, a dual-CAN bus can beimplemented to prevent a bus failure from causing the UAV to losecontrol.

The mission planner 200 is a module that provides the other modules ofthe UAV 102 with high-level directives and goals; the execution of thesedirectives and goals causes the UAV 102 to fulfill a service request.The goals and directives produced by the mission planner 200 arecommunicated to the other modules of the UAV 102, which may then takeother actions to complete a mission, including the generation ofadditional directives and goals for other modules of the system.

For instance, in one embodiment, the mission planner 200 determines aset of waypoints that the UAV 102 may traverse in order to reach adestination site 105, and provides the location of a first waypoint tothe flight controller 201 as a goal, along with a directive to fly tothat location. In this embodiment, the flight controller 201 may then,in turn, compute the orientation and propulsion needed to move the UAV102 towards the goal location; the flight controller 201 may alsogenerate further directives for other modules, such as, for example, forthe actuator control system 204 and for the propulsion management system205. The directives sent to the actuator control system 204 and thepropulsion management system 205 may cause them to take actions thatchange the orientation of the UAV 102 and propel it towards the goallocation. As a result of the actions taken by various modules in the UAV102 in response to the directives and goals of the mission planner 200,the UAV 102 will fly to the designated first waypoint. Once that goal isachieved, the mission planner 200 may send new goals and directives tothe other modules, such that the UAV 102 flies to a second waypoint, anda third waypoint, and so on, until the higher-level goal of reaching thedestination site 105 is fulfilled.

Besides movement directives, the mission planner 200 may issue otherdirectives to the modules of the UAV 102 that cause actions such asdropping of a payload, capturing of image data, transmitting of data,etc. The mission planner 200 may also receive commands from the globalservices 103, from human operators, or from third-party controllers(such as air traffic controllers), and may issue directives to the UAV102 modules based on these commands. For instance, in one exampleembodiment, the mission planner 200, on board a UAV 102, may receive acommand from a human operator to fly back to a distribution center 101due to an approaching storm. In response to this command, the missionplanner 200 will produce new goals and directives that are sent to othermodules in the UAV 102, and as a result of these new goals anddirectives, the UAV 102 will change course and return to thedistribution center 101.

In one embodiment, the mission planner 200 comprises a finite statemachine 208. The finite state machine 208 is a data structure thatorganizes when and under what circumstances the mission planner 200issues goals and directives to the other components of the UAV 102,during the course of the UAV 102's mission. Conceptually, the finitestate machine 208 comprises a plurality of vehicle states andcorresponding valid transitions between those states. At least one ofthe vehicle states is active at all times during the UAV 102's mission.The mission planner 200 broadcasts goals and directives, over thecommunications bus, to the other modules of the UAV 102, based on thecurrent vehicle state. The finite state machine 208 transitions from onevehicle state to another vehicle state as the mission progresses, andwhen the finite state machine 208 enters a new vehicle state, themission planner 200 may broadcast new goals and directives to the othermodules of the UAV 102. For example, in one embodiment, the UAV 102includes the vehicle states: launch, nominal flight, hold position,deliver package, return, and landing. In this embodiment, the missionplanner 200 may begin the mission in the launch state. In the launchstate the mission planner may give the flight controller 201 the goal ofmaking the UAV 102 take off. Based on that goal, the flight controller201 may increase the thrust provided by the engines and may lower theflaps on the wings by issuing directives to the actuator control system204 and the propulsion management system 205. Once the vehicle isairborne, the finite state machine 208 may transition to the nominalflight state. In the nominal flight state, the mission planner 200 maysend the flight controller 201 a directive to fly to a particular goaldestination. Once the UAV 102 reaches the destination, the finite statemachine 208 may transition to the deliver package state. Based on thedeliver package state, the mission planner 200 may send directives toboth the flight controller 201 and the payload management system 206,such that the destination site is safely approached, and the payload isreleased.

The finite state machine 208 may be represented using a variety ofdifferent data structures and can be implemented using a variety ofhardware, software, or hybrid hardware-software methods. In oneembodiment, the finite state machine 208 is implemented by creating atechnical specification defining the vehicle states and valid statetransitions, and then compiling the technical specification to producean executable or object code that represent the defined states andtransitions. In this embodiment, the executable or object code can bestored in a computer storage medium—such as random access memory, harddisc storage, flash memory—in the UAV 102. In another embodiment thetechnical specification may be translated into a hardware design thatcan be implemented using one or more hardware modules.

The mission planner 200 is provided with mission data prior to thelaunch of the UAV 102 from the distribution center 101. The mission dataincludes information that enables the mission planner 200 to locate thedestination site 105, to determine an appropriate route to thatlocation, and to perform any request-specific actions required tocomplete the service request. For example, in some embodiments, themission planner 200 is provided with a destination location, a route tothe destination location, and a series of points along the route whereimages are to be captured with an on-board camera.

In some embodiments, the mission data includes a local skymap for anarea of operation. The area of operation is a geographic region thatencompasses the distribution center 101 and the destination site 105.The local skymap is organized data that includes information that isuseful in mission planning, including the locations of destination sites105, flight corridors, and controlled zones. The local skymap may bestored in a custom data structure, or it may be stored using standarddatabase software such as MYSQL, POSTGRESSQL, MONGODB, COUCHDB, etc.Database software that has the capability to organize and retrieve databased on geographic coordinates is especially suitable for storing theskymap.

A flight corridor is an area of airspace that is designated by the UAS100 for UAV flight. The local conditions in a flight corridor may bemonitored by the UAS 100, and the flight corridors may be used by theUAVs 102 to travel safely and efficiently between locations. The localskymap comprises information about each of a plurality of flightcorridors. The information about each flight corridor may include, butis not limited to, data about the flight corridor's location, altitude,local wind conditions, local air traffic (i.e. other UAVs and aircraftwithin the flight corridor), precipitation, aerial hazards, geographicobstacles (e.g. Mountains), etc. Flight corridors may be subdivided in away that allows them to safely accommodate several aircraft flying inseparate lanes within a single flight corridor. For example, multipleaircraft may be accommodated in a single flight route by assigning eachaircraft a distinct and exclusive lane in the corridor. In oneembodiment, each flight corridor may be sliced horizontal into layers,with vertical separations between the layers, and each layer in thecorridor may be reserved as a lane for a specific UAV 102 for theduration of its mission. For instance, a flight corridor may be slicedinto three layers, one lane from 100 m Above Ground Level (AGL) to 110 mAGL, another lane from 110 m AGL to 120 m AGL, and a third lane from 120m AGL to 130 m AGL. Aircraft may be assigned to these layers (lanes)such that, even if they cross paths in a flight corridor, as long asthey stick to their assigned lanes, their altitude difference ensuresthat a collision will not occur. Prior to taking off from a distributioncenter 101, a UAV 102 may be provided with information that reserves ita slot in one or more flight corridors, which it can safely occupy forthe duration of its mission. These reserved lanes in the flightcorridors can provide a route for the UAV 102 from the distributioncenter 101 to the destination site 105.

A flight corridor may be represented in a skymap in a variety ofdifferent ways. In one embodiment a flight corridor is represented as asequence of geographic coordinates with altitude, stored in a computerstorage medium. For example, a flight corridor could be represented as asequence of latitude, longitude, and altitude (i.e. three-dimensional)points. In this embodiment the sequence of points define a series ofconnected line segments that form a flight corridor. The latitude andlongitude number may be given as GPS coordinates. Additional meta datacan also be associated with the points or the line segments, torepresent additional information about the flight corridors, such as,for example, local weather conditions. In another embodiment the flightcorridors are represented as a sequence of three-dimensional curves(such as Bezier splines), where the curves are defined such that theypass through specific latitude, longitude, and altitude coordinates.

A controlled zone is a geographical area where aircraft may come intoclose proximity with other aircraft, and where, therefore, coordinationbetween aircraft is needed to avoid collisions. The location ofdistribution centers 101 and destination sites 105 are typicallycontrolled zones, since multiple UAVs 102 may attempt to approach theselocations at the same time. Other locations may also be controlledzones; for example, a place where multiple flight corridors intersectmay be a controlled zone. The mission planner 200 of a UAV 102 may takespecific actions when approaching, entering, or leaving a controlledzone, in order to ensure that the UAV 102 maintains a safe distance fromother aircraft. For example, the mission planner 200 may cause thefinite state machine 208 to transition to a new vehicle state.Similarly, the mission planner 200 may cause the UAV 102 to changespeed, altitude, and/or heading (either as a result of transitioning toa new vehicle state, or without a change of vehicle state).

Controlled zones may be represented in a variety of ways. In oneembodiment, a controlled zone is represented by a latitude and longitudecoordinate (a center point for the controlled zone) and a radius. Inthis embodiment the controlled zones are circular in shape. In anotherembodiment a controlled zone is represented by a polygon defined by asequence of latitude and longitude points. For example, a controlledzone may be represented by three GPS coordinates that define atriangular zone, or a controlled zone may be represented by four GPScoordinates that define a rectangular zone. Controlled zones, likeflight corridors, may also include altitude values to indicate thevertical bounds of the zone. For instance the latitude and longitudecoordinates used to define a controlled zone, may also include analtitude coordinate.

In some embodiments, the local skymap is generated from a global skymap,which contains information about a wider geographic area, by selectingthe information in the global skymap that pertains to the area ofoperation. If the global skymap is not excessively large, the entireskymap may be store on the UAV 102, instead of just the local subsetpertaining to the area of operation.

Using the information in the skymap, the mission planner 200 develops adynamic route from the distribution center 101 to the destination site105, prior to launch or soon after launch. The dynamic route takes intoaccount the goals of the mission as well as the requirement of the UAV102 to return to a distribution center 101 after fulfilling the servicerequest. In some embodiments, the mission planner 200 receives apre-generated route from the distribution center 101 or the globalservices 103, and modifies that route only if it receives a command todo so, or if conditions in the area of operations change over time.

The dynamic route is a sequence of flight corridors that the UAV 102 maytraverse to fly from its present location to some goal location. As theUAV 102 flies its mission, it may receive updates to the skymap from theUAS 100, including updates concerning local conditions of the flightcorridors in the area of operation. The updates may be received from theglobal services 103, from the distribution centers 101, or from otherUAVs 102. In some embodiments, updates may also be received from theservice requestors 104, or from third-parties, such as weatherinformation providers, news services, air traffic controllers,satellites, civil aviation authorities, law enforcement, militaryaviation authorities, etc.

The mission planner 200 may modify the dynamic route during the missionas the flight corridor updates are received. For example, in someembodiments, the mission planner 200 may alter the dynamic route toavoid flight hazards such as inclement weather, aircraft trespassinginto a flight corridor, etc. When the dynamic route is modified, themission planner 200 will re-determine the sequence of flight corridorsthat will be traversed to reach the goal location.

As illustrated in FIG. 2D, the UAV 102 also includes a flight controller201. The flight controller 201 provides the mission planner 200 withguidance, navigation, and control functions. For example, the missionplanner 200 is required to know the location, orientation, altitude, andspeed of the UAV 102 at various times during the mission, and the flightcontroller 201 provides this information through a process called stateestimation. Similarly, when the mission planner 200 requires the UAV 102to move from one point to another, it sends commands to the flightcontroller 201 to achieve that goal. The flight controller 201communicates over the bus with the sensor system 202, the actuatorcontrol system 204, and the propulsion management system 205, to providethe guidance, navigation, and control functions.

The sensor system 202 provides information from sensor instruments tothe flight controller 201. In some embodiments, the sensor system 202comprises several instruments, such as, for example, a GPS unit, anInertial Measurement Unit (IMU), dynamic pressure sensor, staticpressure sensor, air temperature reader, etc.

The actuator control system 204 includes actuators that control variousmovable parts on the UAV 102, including the control surfaces 213 on theairframe. The actuators may comprise of electrical servos, hydraulics,or any other technology that can be used to enable mechanical movement.The actuator control system 204 can change the state of the actuatorsbased on commands from the flight controller 201. The actuator controlsystem 204 may also have a means of measurement (e.g. an encoder,resolver, or potentiometer) that can give feedback on the positions andmovements of the control surfaces 213. The positions and movements ofthe control surfaces 213 may be reported back to the flight controller201.

Since any moving mechanical system may develop small inaccuracies,defects, or “drift” as a result of the manufacturing process or as aresult of use, the control surfaces 213, the actuators, and the means ofmeasurement, must be periodically calibrated so that the real positionsand motions of the control surfaces 213 correspond closely to thepositions and motions that the actuator control system 204 receives fromthe means of measurement. The systems and methods used to perform thiscalibration are described in more detail in connection with FIG. 5 .

The propulsion management system 205 controls the force exerted by theengines mounted on the UAV 102—for example by adjusting the speed ofpropellers mounted on a propeller powered UAV—and monitors the amount ofbattery capacity remaining on the UAV. The flight controller 201 canadjust the speed of travel of the UAV 102 by communicating with thepropulsion management system 205. The propulsion management system 205may also monitor the physical attributes of batteries onboard the UAV102, such as changes in a battery's voltage over time, changes in thecurrent output by a battery over time, and even changes in a battery'scell chemistry over the duration of a flight. The monitored informationmay be communicated to the distribution center 101 or global services103, either during the mission via the communications system 203, orafter the UAV 102 has returned to a distribution center 101. In thelatter case the information may be communicated by wireless means, wiredmeans, or by physical transfer of some storage medium such as a memorycard.

The flight controller 201 receives information from the sensormanagement system 202 and the actuator control system 204, and performsa state estimation that provides a best guess of the UAV 102's position,orientation, and speed to the mission planner 200. The state estimationis continuously updated and checked as the various systems of the UAV102 provide new information.

The mission planner 200 determines the high-level goal location that theUAV 102 must travel to and communicates the goal location to the flightcontroller 201. The mission planner 200 may communicate directives andgoals to the flight controller 201 using any appropriate technique(s).For example, in one embodiment, the mission planner 200 communicatesmovement goals to the flight controller 201 via a sequence of waypoints.In another alternative embodiment, the mission planner 200 communicatesmovement goals to the flight controller 201 via splines.

The flight controller 201 receives the movement goals—as waypoints,splines, or any other suitable form—and determines, based on rules orphysics-based models, the commands that must be communicated to theactuator control system 204 and the propulsion management system 205 toachieve the movement goals. For example, according to some embodiments,the physics-based models output the required ruddervator and wing flapstate, and the engine thrust for the UAV 102, based on the current stateestimation (i.e. the UAV 102's position, orientation, and speed), andthe local conditions including wind and temperature.

The communication system 203 comprises transmitters and receivers thatenable the UAV 102 to send and receive information using differentcommunications protocols. The communication system 203 may includetransmitters and receivers for standard cellular radio technologies suchas CDMA, GSM, 3G/4G, LTE, etc., as well as line-of-sight radios thatallow the UAV 102 to directly communicate with a distribution center 101or another UAV 102.

The communications system 203 may also support mesh networking such thata first UAV 102 can receive and rebroadcast a signal sent by a secondUAV 102, in order to increase the effective communications range of thesecond UAV 102. A third UAV 102 may similarly receive and rebroadcastthe signal that was rebroadcast by the first UAV 102, increasing thecommunications range further still. This method of receiving andrebroadcasting signals for other UAVs may be implemented across theentire area of operations for the UAS 100 to enhance the connectivity ofevery UAV 102 operating in the system. A custom mesh protocol may beused to implement the mesh networking, or a standard mesh networkingprotocol such as IEEE 802.15.4, ZIGBEE, BATMAN, HWMP, IEEE 802.11 MESH,etc. In one embodiment of the system, only the line-of-sight radio isused to implement the mesh networking protocol.

The communications system 203 is used by the mission planner 200 toperiodically broadcast status messages that contain information aboutthe UAV 102's current status. The purpose of the broadcasted statusmessages is to inform the distribution centers 101, the global services103, and other UAVs 102 about important information concerning the UAV.For example, a status message may contain information about a UAV 102'scurrent position, current heading, current speed, current destination,etc. Other information, such as a UAV 102's payload, status of itsbatteries, and other similar information may also be broadcast. Somespecial information related to air traffic management may also bebroadcasted in a status message, and this use of status messages isdescribed in more detail below. A status message broadcasted by a UAV102, may be rebroadcast by other UAVs 102 based on a mesh networkingprotocol, in order to increase the status message's effective range.

The payload management system 206 performs various functions related tothe payload carried by the UAV 102, depending on the nature of theservice request and the payload. For example, when the payload isattached to the UAV 102 prior to launch, the payload management system206, will communicate that the attachment is successful to the missionplanner 200 and/or the distribution center 101. In the case where theservice request is a package delivery, the payload management system 206also monitors the state of the payload—for example the temperature ofthe payload in the case where the payload is perishable—and manages therelease of the payload at the destination site 105. In this example, themission planner 200 determines the location, altitude, speed, andorientation of the UAV 102 required to drop the payload safely at thedestination site 105, and communicates a command to release the payloadat the appropriate time to the payload management system 206. Thepayload management system 206 receives the command and causes thepayload bay doors 215 to open, dropping the payload.

The payload management system 206 may perform other functions dependingon the nature of the payload. For example, in the case where the servicerequest is related to surveillance or mapping, the payload managementsystem 206 may interface with a camera system included in the payloadand can capture images or video based on instructions received from themission planner 200. For instance, in this embodiment, the missionplanner 200 may issue a command to the payload management system 206 tocapture images when the UAV 102 flies over some point of interest in itsroute.

The safety system 207 manages various failsafe components mounted on theUAV 102. For example, in one embodiment, the safety system 207 monitorsand controls a parachute system that may be deployed based on a commandreceived from the mission planner 200, or based on information receiveddirectly from the flight controller 201 or sensor system 202. Forinstance, if the UAV 102 enters a non-recoverable dive, the safetysystem 207 may deploy the parachute based on data received from thesensor system 202. In another embodiment, the mission planner 200 mayinstruct the safety system 207 to deploy a parachute based on a messagereceived from the global services 103 or a distribution center 101.Parachute deployment on command may be useful in situations where an airtraffic control process detects the possibility of imminent collisionbetween multiple aircraft in an area with heavy air traffic. Forcing aUAV 102 to deploy its parachute and descend may prevent it from enteringthe flight path of other aircraft.

The mission planner 200 is designed to follow an air traffic managementprotocol to minimize the risk of collisions between UAVs 102 operatingin the system. The air traffic management protocol defines actions thatmust be performed and messages that must be broadcast by UAVs 102 atspecific times in a mission and/or under specific conditions.

The air traffic management protocol provides rules that the aircraftwill follow as it traverses the skymap. The rules define specificmessages that must be broadcast based on an aircraft's location andcircumstances, as well as based on the location and circumstances ofother aircraft. The rules also similarly specify actions that must beperformed by an aircraft, based on its circumstances, the circumstancesof other aircraft in the area, and based on the messages it receivesfrom other aircraft. The rules specified by the air traffic managementprotocol are put into practice by the software/hardware systems andmethods in the UAV 102, including the systems described above, such asthe flight controller 201, the sensor system 202, and the communicationssystem 203. The mission planner 200 is the hub where information aboutthe UAV's circumstances is aggregated, including communicationsreceived, location in the skymap, flight information, etc. The missionplanner 200 issues commands to the other modules and systems of the UAV102, based on the aggregated information and the air traffic managementprotocol.

Although the UAV 102 is designed to operate autonomously, the missionplanner 200 is configured to receive instructions via the communicationssystem 203 that may override the mission planner 200's flight plans. Forexample, the UAV 102 may receive instructions from a distribution center101 or the global services 103 that command the UAV 102 to return tobase immediately due to bad weather or a passenger aircraft entering thearea. On receiving such a command the mission planner 200 will changethe movement goals of the UAV 102 and issue new directives to the othermodules, so that the UAV 102 adjusts its flight path as necessary. TheUAV 102 may continue to apply the air traffic management protocol to itsflight along the adjusted route. The structure and functionality of theUAV 102 described above has been divided into modules based on oneexample implementation, but the functionality of various modules may bemerged or further split such that there are more or less components thanhave been illustrated in FIG. 2D. It is also possible to devolve some ofthe functionality of the various modules directly into the servos,sensors, and other hardware components of the UAV 102. For instance, theflight controller 201 may communicate directly with a plurality of servomotors, each of which has the functionality of the described actuatorcontrol system 204. Such a decentralization of hardware componentcontrol may be beneficial in some implementations from the point of viewof fault-tolerance.

Distribution Center

The distribution center 101 handles the local logistics for the UAS 100.When the global services 103 receive a service request from a servicerequestor 104, the global services 103 will select a distribution center101 to fulfill the service request according to criteria in the servicerequest, including the location of the destination site 105. The globalservices 103 will then send at least a portion of the information in theservice request to the selected distribution center 101.

The distribution center 101 is responsible for launching and recoveringUAVs 102, maintaining and monitoring inventories of payloads, equipment,batteries, and UAVs 102, and communicating local information to theglobal services 103. Other functions such as UAV, battery, and payloadselection for missions, mission data preparation, UAV monitoring andcommunication during the mission, etc., can be performed by either thedistribution centers 101 or the global services 103, depending onimplementation and/or system status. A distribution center operator 107may be stationed at the distribution center 101 to facilitate thedistribution center operations.

FIG. 3 is a block diagram of a distribution center 101, according to oneexample embodiment. As mentioned previously, some of the functionsperformed by this embodiment of the distribution center 101 could beperformed by the global services 103 instead. Similarly, some of thefunctions of the global services 103 could be performed locally by thedistribution center 101. System designers with skill in the art maydivide the functionality of the global services 103 and the distributioncenters 101 in any appropriate way based on the requirements of aparticular UAS implementation.

In this embodiment, the distribution center 101 is comprised of abattery management system 301, a payload inventory management system302, a UAV operations system 303, a distribution center managementsystem 304, an operator interface 312, and a UAV inventory managementsystem 313.

The distribution center management system 304 serves as the hub of thedistribution center 101. In this embodiment, the distribution centermanagement system 304 comprises a mission manager 305, sensor station306, communications station 307, logistics system 308, skymap database309, terrain map database 310, and interface handler 311. In one exampleembodiment, the distribution center management system 304 is implementedusing one or more computer servers that have specialized sensor andcommunications peripherals installed.

Some of the functions of the distribution center 101 may require theassistance of a human distribution center operator 107. For example, UAVassembly, UAV repair, payload attachment and detachment, UAV recovery,battery replacement, placing UAVs 102 on a UAV launcher, and placingbatteries into charging equipment are tasks that may require humaninvolvement if they are not fully automated. The operator interface 312allows the distribution center operator 107 to receive information andinstructions from the distribution center management system 304 and theglobal services 103, as well as to send information and instructionsback to the distribution center management system 304 and the globalservices 103. The distribution center management system 304 communicateswith the operator interface 312 via the interface handler 311. In someembodiments, the operator interface 312 is an application running on asmartphone, a tablet computer, or a personal computer, and the interfacehandler 311 communicates with the application via a wirelesscommunications protocol, such as IEEE 802.11.

The mission manager 305 is a module that is responsible for managing thelocal aspects of mission operations at the distribution center 101. Insome embodiments, the mission manager 305 receives service requests (ordata derived from the service requests) from the global services 103,selects a UAV 102 or UAV components that will be assembled into a UAV102, prepares the mission data that will be utilized by the UAV 102during the mission, selects an appropriate payload for the mission,tests and launches the UAV 102, and monitors the status of the UAV 102and payload during the mission. The mission manager 305 communicateswith the distribution center operator 107 via the operator interface 312during various stages of the mission to communicate both the status ofthe mission, as well as instructions indicating the actions to be takento facilitate the preparation, loading, launch, and recovery of UAVs102.

The mission manager 305 utilizes the other components of thedistribution center management system 304 to monitor the status of thelocal environment and various local components of the UAS 100, includingthe UAVs 102, UAV operations system 303, battery management system 301,and the local inventories.

The mission manager 305 maintains contact with the global services 103and local UAVs 102 through the communications station 307. Informationabout service requests is received from the global services 103, andinformation about local conditions, ongoing missions, inventories, etc.,is transmitted back to the global services 103. The communicationsstation 307 may have transmitters and receivers for several differentstandard telecommunications protocols, including, but not limited toGSM, CDMA, GSM, 3G/4G, LTE, etc. In addition to these standardprotocols, the communications station 307 may also support line-of-sightor mesh protocols to enable direct communications with UAVs 102 andother distribution centers 101. Finally, the communications station 307may also include a wired connection to the Internet for high-speedcommunication with the other components of the UAS 100 and third-partyinformation providers. The mission manager 305 may send some of theinformation received via the communications station 307 to the operatorinterface 312, so that the distribution center operator 107 can monitorthe status of UAVs 102 or other components of the UAS 100 that arerelevant to a local mission.

The sensor station 306 is primarily used to gather local weather datafor the distribution center 101's location. The sensor station 306 mayinclude pressure sensors, thermometers, wind sensors, precipitationdetectors, etc. The sensor station 306 may also be used to detect andtrack UAVs 102 using instruments such as radars, radio trackers, andoptical object recognition systems. The mission manager 305 may presentinformation from the sensor station 306 to the distribution centeroperator 107 via the operator interface 312, so that the distributioncenter operator 107 can take actions necessary to protect the UAVs 102and distribution center 101 from inclement weather. For example, if thesensor station 306 detects an approaching storm, the mission manager 305may display a notification to the distribution center operator 107 viathe operator interface 312, and the operator 107 can follow proceduresto recover UAVs 102 that have already been launched, to abort missionsthat have not been launched, and the like.

Prior to any vehicle mission being launched, the mission manager 305determines the mission energy requirement for the mission based onmission information. Mission information may include information fromthe service request, information about the local conditions (such astemperature, wind, and precipitation), and information about thecharacteristics of the vehicle and payload. For example, in oneembodiment, the energy requirement is determined by computer analysis offactors such as the location of the destination site 105, the route tothe destination site 105, the wind conditions along the route to thedestination site 105, the flight characteristics of the UAV 102 thatwill handle the mission (such as its typical power consumption per unitdistance flown, aerodynamic drag, etc.), and the weight of the payloadthat is needed to fulfill the service request. The mission manager 305communicates the mission energy requirement to the battery managementsystem 301, which then selects one or more batteries that can providesufficient power for the mission, from the battery inventory. Thebattery management system 301 may automatically install the selectedbatteries into the UAV 102 that will fly the mission, or it may useindicators to notify the distribution center operator 107 that specificbatteries must be installed. Further details related to the operation ofthe battery management system 301 are discussed below.

The logistics system 308 tracks the inventory levels of variouscomponents at the distribution center 101, and reports these inventorylevels to the global services 103 and the mission manager 305. Thisinventory information may be used when selecting a particulardistribution center 101 to fulfill a service request.

The logistics system 308 interfaces with the battery management system301, the payload inventory management system 302, and the UAV inventorymanagement system 313 to determine the inventory levels of batteries,payloads, and UAVs/UAV components, respectively. The logistics system308 is capable of requesting deliveries of additional stock wheninventory levels fall below a threshold level or when inventory levelsare predicted to fall below a threshold level within a specified periodof time.

The global services 103 may monitor the inventory levels reported by thelogistics system 308 and may proactively dispatch additional inventoryitems to a distribution center 101 based on current inventory levels orthe predicted inventory levels in the future. The logistics system 308may also directly notify the distribution center operator 107 ofinventory shortages, or of errors with specific items in an inventory,via the operator interface 312. Based on these notifications, thedistribution center operator 107 may restock or repair items as needed.

Each item at the distribution center 101 may be affixed with a trackingtag that can be monitored by the logistics system 308. Varioustechnologies can be used to implement the tracking tags, including barcodes, RFID tags, NFC tags, etc. These tags may be affixed to every itemat the distribution center 101 that requires tracking, including UAVs102, UAV components, payloads, batteries, spare parts, etc. The tagsassociate an object identifier with each tracked physical object at thedistribution center 101. For example, each payload at the distributioncenter 101 will have an object identifier associated with it that isindicated by the tag affixed to it. The object identifier may be readfrom the tag by way of a reader that is configured to scan the tag. Forexample, an RFID tag would be read using an RFID reader, an NFC tagusing an NFC reader, etc.

The object identifiers can be used to determine the type of an objectthat has been scanned as well as its unique identity. For example, thetag affixed to a payload object will identify that the object is apayload of a specific type, as well as that it is a specific instance ofthat payload, different from other payloads of the same type in theinventory. In some embodiments, the object identifier can be used todetermine a database entry associated with the object in an inventorydatabase. The logistics system 308 reports the inventory levels forobjects associated with each object identifier to the global services103.

The illustrated embodiment of a distribution center 101 also includes apayload inventory management system 302 that tracks inventory levels andstatus for various payloads that may be mounted to the UAVs 102. Thepayload inventory management system 302 may also provide recharging,refrigeration, and other maintenance related functions related tospecific payloads. For instance, if the payload is a vaccine, then thepayload inventory management system may provide a refrigerated storagecontainer for vaccine doses and may monitor and report the temperaturein the storage container and the number of doses stored in the containerto the logistics system 308. The mission manager 305 may notify thedistribution center operator 107 of the status of various payloadsstored at the distribution center 101 via the operator interface 312.For example, in some embodiments, the mission manager 305 may send anotification to the operator interface 312 to notify the distributioncenter operator 107 that a particular vaccine stored in the payloadinventory management system 302 has expired. Based on this notificationthe distribution center operator 107 may remove the expired vaccine fromstorage and replace it with new stock.

The UAV inventory management system 313 tracks the assembled UAVs 102and UAV components stored at the distribution center 101, and reportsthis information to the logistics system 308. The mission manager 305 orglobal services 103 may query the logistics system 308 to determine theUAV resources available for missions at a particular distribution center101, and may allocate these resources based on the requirements ofservice requests received by the UAS 100. When a particular UAVconfiguration is required to fulfill a mission, the mission manager 305may send instructions to the distribution center operator 107, via theoperator interface 312, to assemble a particular set of UAVcomponents—stored in the UAV inventory management system 313—toconstruct a UAV suitable to complete that mission. As UAVs or UAVcomponents are added and removed from the inventory, the UAV inventorymanagement system 313 tracks not only the availability of theseresources, but also the status of components, such as their conditionand need for replacement. This information may be used by the missionmanager 305 and the global services 103 to order deliveries of new UAVs102 or components for the distribution center 101.

The mission data that will be uploaded to the UAV 102 is prepared basedon the requirements of the service request received from the globalservices 103. Although we discuss the preparation of the mission data bythe mission manager 305, alternative embodiments are possible, whereeither the global services 103 or the mission planner 200 onboard theUAV 102, prepare the mission data. The mission data includes not onlythe location of the destination site 105 and the payload required tosatisfy the service request, but also information required to generate aflight route to the destination location. The information required forroute generation is stored locally in the skymap database 309 and theterrain map database 310.

The skymap database 309 contains data about a plurality of flightcorridors in the geographic region served by the distribution center101. The skymap database 309 may be at least partially synchronized witha global skymap database 400 that is part of the global services 103.The flight corridor data includes information about the real-timeconditions within the flight corridors, such as weather, air traffic,etc. The local skymap database 309 updates the flight corridor databased on the latest information received from the global services 103,other distribution centers 101, and third parties (such as weatherservices and air traffic controllers). UAVs 102 that have recently flownin a flight corridor may also send data to the distribution center 101about the last monitored conditions present in the flight corridor, andthis information may be used by the skymap database 309 to update therelevant flight corridor data. When the local skymap database 309 at thedistribution center 101 has more recent information about a flightcorridor than the global skymap database 400, the global skymap database400 is updated via the communications station 307. The reverse is alsotrue, and the latest updates from the global skymap database 400 arereceived via the communications station 307 and incorporated into thelocal skymap database 309.

The terrain map database 310 contains terrain data, which is informationabout the terrain and ground obstacles in the geographic region servedby the distribution center 101. This terrain data can be stored in anumber of ways, including but not limited to, as raw images, as aheightmap, and as a three-dimensional (3D) mesh. The global services 103also include a global terrain map database 401, which is at leastpartially synchronized with the local terrain map database 310. As inthe case with the skymap databases, the terrain map database 310 isupdated based on data captured from UAVs 102 during their missionflights. For example, if a UAV 102 flies over a location and capturesinformation regarding a new obstacle that was not present at thatlocation in the terrain map database 310, the terrain map database 310will be updated with the new information via data received from the UAV102, either during the mission, or after the UAV 102 has been returnedto the distribution center 101.

Although the information about the flight corridors from the skymapdatabase 309 may be sufficient to route the UAV 102 to the destinationsite 105, information about the ground that the UAV 102 is flying overcan also be useful during various phases of the mission. For instance,during UAV launch and recovery, the terrain and obstacles near thelaunch and recovery sites are relevant. In addition, if the servicerequest requires a package delivery, then the terrain and obstacles atthe destination site 105 are relevant, as the UAV 102's mission planner200 must determine a location from which to drop the payload such thatthe payload lands in an accessible place and does not damage localstructures, objects, or persons.

The information from the terrain map database 310 is also useful forfulfilling service requests that require surveillance or mapping. Insome instances, the terrain data from the terrain map database 310 canbe used to fulfill a surveillance or mapping request without launching aUAV. For example, if a UAV 102 has recently captured imagery at adestination site 105 at a particular location, and a subsequent servicerequest asks for image capture at the same location, within somethreshold time limit, then the most recent information from thedestination site 105 that has been stored in the terrain map database310 can be sent to the service requestor 104.

To prepare the mission data locally, the mission manager 305 firstdetermines the location of the destination site 105 from the servicerequest information received from the global services 103. Based on thisdestination location, and the launch location, which is typically thelocation of the distribution center 101, the mission manager 305determines the relevant area of operations for the mission, and extractsthe data associated with this geographic region from the skymap database309 and the terrain map database 310. The extracted information is sentto the UAV 102 as part of the mission data. In some embodiments, themission manager 305 also provides the UAV 102 with a lowest cost routeto the destination site 105 as part of the mission data. Depending onthe implementation, the route can be dynamically updated by the globalservices 103, the mission planner 200 in the UAV 102, and/or the missionmanager 305. When the connectivity to the UAV 102 cannot be guaranteedduring the mission flight, the mission planner 200 onboard the UAV 102may be allowed to dynamically update the route.

In some embodiments, the UAV 102 stores complete mirrors of the skymapdatabase 309 and the terrain map database 310, instead of only subsetsof the information in these databases. This can be done when the size ofthe databases is small enough that the storage resources on the UAV 102are sufficient to store the entire dataset. When this is not the case, asubset of the information may be stored in the UAV 102, as describedearlier. Similarly, in the case where the local skymap database 309 andlocal terrain map database 310 have sufficient storage capacity, theentire global skymap 400 and global terrain map 401 may be storedlocally at the distribution center 101. Subsets of the global data maybe extracted and stored locally only when the global data sets are toolarge for complete local mirroring to be economical.

The UAV operations system 303 is responsible for testing and calibrationof the UAVs 102, launching of UAVs 102, and recovery of UAVs 102. TheUAV operations system 303 may comprise three separate components: atesting and calibration system, a UAV launcher, and a UAV recoverysystem. However, it is possible to combine two or more of thesecomponents into a single integrated device to improve speed andefficiency. For example, the testing and calibration system may becombined with the UAV launcher so that the aircraft is verified andtested as it sits on the launcher.

The testing and calibration system tests the components of the UAVs 102to ensure that they will perform during the mission. Faulty componentsare identified during testing and are brought to the attention of thedistribution center operator 107 via the operator interface 312. Thetesting and calibration system also verifies, via the affixed tags, thateach component in the assembled UAV 102 is a component allocated by themission manager 305 for the current mission. For example, the testingand calibration system detects the battery and engines attached to theUAV 102, and ensures that they have adequate charge and performance forthe mission, prior to launch. Discrepancies are brought to the attentionof the distribution center operator 107 for correction. Similarly, thetesting and calibration system verifies that the payload loaded onto theUAV 102 is the right payload for the current mission. Finally, thetesting and calibration system also tests the control surfaces and othermoveable components of the UAV 102, to ensure that these componentsrespond in the expected manner to commands sent by the mission planner200 and flight controller 201. The testing and calibration system isdescribed in more detail in connection with the description for FIG. 5 .

The UAV launcher is an automated system that is used to launch the UAVs102 in an efficient and reliable manner. The UAV launcher and thetesting and calibration system may be combined into a single device.

The UAV recovery system is an automated system that is used to recoverthe UAVs 102 in a manner that minimizes the wear and tear on thefuselage of the UAVs, while also minimizing the risk to human operatorsin the vicinity.

The UAV operations system 303 is designed to allow the distributioncenter operator 107 to launch and recover many aircraft reliably, withina short period of time.

Once the UAV 102 is launched, and the mission manager 305 continues tomonitor the aircraft during the mission flight. The mission manager 305receives status updates from the UAV 102, and these status updatesenable the mission manager 305 to track the progress of the mission atleast intermittently. The mission manager 305 may present informationrelated to the UAV 102's status to the distribution center operator 107via the operator interface 312. In the event that there is some localevent that requires a mission termination, such as, for example, anapproaching storm, either the mission manager 305 or the distributioncenter operator 107 (or both), via the operator interface 312, can senda command to the UAV 102, through the communications station 307, toinstruct the UAV 102 to return to the distribution center 101.

Global Services

The global services 103 are a collection of software services running onone or more computer servers, accessible through the Internet or anothercommunications protocol. In one example embodiment, the global services103 are software modules running on virtual machines in a third-partydata center, such as Amazon Web Services or Google Cloud.

One purpose of the global services 103 is to provide a globalinfrastructure to coordinate, support, and manage multiple distributioncenters 101, service requestors 104, and UAVs 102. However, in someembodiments, it is conceivable that the functionality of the globalservices 103 is provided by a local computer server, and that the serverserves a local set of UAVs 102, distribution centers 101, and servicerequestors 104—possibly only a single one of each.

One or more global system operators 106 and remote vehicle operators 108connect to the global services 103 and provide human intervention forsystems that cannot be fully automated (or require temporary humanassistance). The global system operators 106 and remote vehicleoperators 108 typically connect to the global services 103 throughcontrol devices. A control device may be a computer workstation, apersonal computer, a tablet device, a smartphone, or any other computingdevice that can communicate through a network with the global services103. For instance, in one example embodiment, a global system operator106 uses a laptop computer, with an Internet connection, to connect tothe global services 103 executing on a computer server, which is alsoconnected to the Internet.

In the example embodiment illustrated in FIG. 4 , the global services103 are configured to serve a plurality of distribution centers 101,UAVs 102, and service requestors 104. In this embodiment, the globalservices 103 comprise a global skymap database 400, a global terrain mapdatabase 401, a data manager 402, a service request handler 403, aglobal logistics manager 404, an air traffic control system 405, asystem interface manager 406, and a remote vehicle manager 407.

As discussed earlier, the global skymap database 400 and the globalterrain map database 401 are global repositories for skymap and terrainmap data in the UAS 100. As was the case with the local versions ofthese databases, the information in these databases can be representedin various ways depending on the needs of the UAS 100. Although thesedatabases are represented as single units in the illustrated embodiment,in practice the databases may be implemented using several mirrored datastores to provide improved read speed, redundancy, and error recovery.

The data manager 402 manages data-writes into, and data-reads out of theglobal databases. For example, as updates to the local skymap databases309 and local terrain map databases 310 are communicated to the globalservices 103, the data manager 402 ensures that the information isstored in the appropriate database and that the latest information isalways available and is not overwritten by out-of-date information. Thedata manager 402 also manages information received from outside of theUAS 100 and integrates this information into the global databases. Forinstance, information received from third-party weather informationproviders, aviation authorities, and external air traffic controllersmay be integrated into the global skymap database 400. Similarly,third-party topographical data, map imagery, and surveillance data maybe integrated into the global terrain map database.

The data manager 402 also manages the updates sent to the localdatabases at each distribution center 101. In one embodiment, as theglobal skymap database 400 and global terrain map database 401 areupdated, the data manager 402 will monitor the regions where thoseupdates are relevant, and will send at least a portion of those updatesto distribution centers 101 that are in the relevant regions. In anotherembodiment, the mission manager 305 at a distribution center 101 in aparticular region will periodically request information about thatregion from the global services 103, and the data manager 402 willdetermine the set of information that is relevant to that region fromthe global databases, and will send that information to the distributioncenter 101, where the information may be integrated into the localdatabases. Similarly, a UAV 102 in flight may request information aboutits current location from the global services 103, and the data manager402 may similarly determine the relevant information that should be sentback to the UAV 102 based on the UAV 102's location.

The service request handler 403 manages service requests sent by servicerequestors 104 that are received by the global services 103. When aservice request is received by the global services 103, the servicerequest handler 403 will communicate with the global logistics manager404 to determine a distribution center 101 that is suitable for handlingthe service request locally. As mentioned previously, the selection ofthe distribution center 101 may take into account not only the locationof a destination site 105 specified in the service request, but also thelogistic requirements of the request, such as payload, UAV capability,etc. For instance, a service request may include information thatspecifies a payload type required to complete the request, and thedistribution center 101 may be selected based on the availability ofthat payload type at various distribution centers 101.

The payload type may be specified directly by means of a payloadidentifier associated with a type of payload, or it may be specified byimplication. For example, a camera payload may be specified byimplication if the service request is a request for image data at thedestination site 105.

In some embodiments, the service request handler 403 takes input from aglobal system operator 106 to determine the distribution center 101 thatwill be used to fulfill a service request.

Once the distribution center 101 and UAV 102 have been identified andreserved to fulfill a service request, the service request handler 403may notify the service requestor 104 that the service request is inprocess. The service request handler 403 may also receive informationfrom the distribution center 101 and/or the UAV 102 that allows apredicted time of mission completion to be estimated and sent to theservice requestor 104.

The service request handler 403 is capable of communicating with theservice requestor 104 via the system interface manager 406. A humanservice requestor 104 will typically send a service request to theglobal services 103 by means of some remote client device such as amobile phone, a tablet, or a personal computer. The system interfacemanager 406 is capable of sending information to the client deviceoperated by the service requestor 104 that is configured to be displayedon the client device. For example, in one embodiment, the systeminterface manager 406 functions as a web server, and the client deviceconnects to the web server and displays a web page that is downloadedfrom the system interface manager 406. In this example, the servicerequestor 104 can receive and send information to the global services103 via the displayed web page. In another embodiment, the systeminterface manager 406 exposes an application interface over the Internet(such as a representational state transfer, or “REST” interface), and anapplication running on the client device is configured to displayinformation received from the global services 103 to the servicerequestor 104, and to send information inputted by the service requestor104 back to the global services 103.

The service request handler 403 may also play an active part indetermining the route a UAV 102 takes on a mission to complete a servicerequest. For example, the service request handler 403 may use the systeminterface manager 406 to query a service requestor 104 for the preciselocation of the destination site 105, and the information provided bythe service requestor 104 may be used to refine the flight route used bythe UAV 102 in fulfilling the service request.

The structure and functionality of the global services 103, describedabove, has been divided into modules based on one exampleimplementation, but the functionality of various modules may be mergedor further split such that there are more or fewer components than havebeen illustrated in FIG. 4 . For example, it is possible to merge theskymap and terrain map databases into a single data store. Some of theservices illustrated can be moved outside the UAS 100, for example, theair traffic control system 405 and the global logistics manager 404 maybe operated outside the UAS 100 as independent services, accessiblethrough an Application Programming Interface (API). These and otherchanges to the structure do not change the overall architecture of thesystem, and systems with such changes may operate in the same or similarmanner as the system disclosed.

The service request handler 403 utilizes the global logistics manager404 to obtain information required for distribution center 101 and UAV102 selection. The global logistics manager 404 tracks the inventoryinformation in each local logistics system 308 at each distributioncenter 101. The global logistics manager 404 may proactively routeadditional stock to local distribution centers 101 when supplies of anyinventory item are depleted, are below some threshold quantity, or arepredicted to be depleted within some threshold time. The globallogistics manager 404 may also notify a global system operator 106 inthe event of an inventory shortage at a distribution center 101. Theglobal system operator 106 may take actions outside the UAS 100 to routenew inventory items to the distribution center 101, such as, forexample, ordering and shipping items from a third-party warehouse.

In one embodiment, the global logistics manager 404 relocates UAVs 102from a first distribution center 101 that has an excess of UAVs to asecond distribution center 101 that has a shortage of UAVs. In thisembodiment, the global logistics manager 404 may monitor the daily,monthly, or yearly patterns of service requests to determine theestimated UAV requirements at each distribution center 101 over a periodof time. Based on these estimated UAV requirements, the global logisticsmanager 404 may preemptively relocate UAVs from one distribution center101 to another. The relocation of UAVs 102 may be done using third-partyshippers, or the relocation may be done by sending requests to thedistribution centers 101 to launch UAVs 102 with destination sites 105set to other distribution centers 101. As an optimization, theserelocation flights may be scheduled during times when the servicerequest volume is low, for example, late at night or during holidays.

The air traffic control system 405 is responsible for tracking the UAVs102 and aircraft that are known to be in flight in the area served bythe UAS 100. The air traffic control system 405 receives informationfrom the distribution centers 101, the UAVs 102 in flight, and fromthird party air traffic information providers. The information receivedby the air traffic control system 405 includes the known positions ofaircraft in the area of the UAS 100, as well as flight routes that areregistered with the system. Distribution centers 101 and/or UAVs 102 mayregister flight routes for missions, with the air traffic control system405. The air traffic control system 405 may also allow UAVs and aircraftoperated by third parties to register their flight routes.

The air traffic control system 405 provides real-time informationupdates regarding the positions of aircraft and UAVs to UAVs 102 thatare flying missions. Using this information, the mission planners 200onboard the UAVs 102 may modify their flight routes to avoid collidingwith other aircraft. The air traffic control system 405 may offersimilar information updates to UAVs and other aircraft that areoperating outside the UAS 100 in order to maintain a safer airspace forall aircraft operations.

The air traffic control system 405 also provides information to theservice request handler 403 and the global logistics manager 404.Information from the air traffic control system 405 may be used toinfluence the selection of distribution centers 101 for service requestsand the relocation of UAVs 102. For example, a service request may berouted away from distribution centers 101 where there is an excess ofair traffic in the vicinity, and UAV relocation may be timed to avoidperiods when air traffic is at its highest.

The remote vehicle manager 407 provides autonomous vehicles, such as theUAVs 102, with a human operator on demand. In the course of a mission, aUAV 102 may encounter a situation that its mission planner 200 or flightcontroller 201 cannot understand or safely handle. For example, anautonomous vehicle in an urban environment may not have the necessaryrouting and vision systems to handle path-determination and objectavoidance in such a complex setting. Thus, an autonomous vehicle, suchas a UAV 102, may fly under the control of its onboard mission planner200 and flight controller 201 while in a relatively simple anduncluttered environment outside a city, but may then request humanassistance once a city is entered.

Testing and Calibration System

The testing and calibration system is an automated system that performsseveral verification functions prior to the launch of the UAV 102. Thetesting and calibration system calibrates the movable parts of the UAV102 to ensure that the parts move as expected when commanded by themodules of the UAV 102 (such as the actuator control system 204, flightcontroller 201, etc.). The testing and calibration system also verifiesthat the UAV 102 comprises the expected components that are necessaryfor the mission, including the correct payload. Finally, the testing andcalibration system may test the engines and other components to detectfailures and defects, and to predict future failures and defects wherepossible.

FIG. 5A illustrates an embodiment of a testing and calibration system500. In the illustrated embodiment the testing and calibration system500 comprises a support stand 501, several cameras 502, fuselage supportstruts 503, and tail support strut 504.

FIG. 5B illustrates the same embodiment of the testing and calibrationsystem 500 with a UAV 102 placed upon it for testing. FIG. 5Cillustrates a side view of an embodiment of the testing and calibrationsystem 500, with a UAV 102 placed upon it.

The support stand 501 is a frame of durable material that acts as asupport upon which the other components of the testing and calibrationsystem 500 can be mounted. The support stand 501 ensures that the othercomponents of the testing and calibration system 500 maintain a fixedposition relative to each other and the UAV 102 after many duty cycles.As described below, the testing and calibration system 500 may use avision-based system for various testing and calibration tasks, and forthese tasks it is helpful that the UAV 102 and the cameras be in apredictable position relative to each other.

The fuselage support struts 503 and the tail support strut 504 providedirect support for the UAV 102 when it is placed on the testing andcalibration system 500. In the illustrated embodiment the UAV 102 isplaced upon the support stand 501 in an inverted position, so that thecontrol surfaces 213 of the UAV 102 are close to the cameras 502 of thetesting and calibration system 500. In an alternative embodiment thesupport stand and support struts may be arranged in a different physicalconfiguration to accommodate a UAV with different physical dimensions.The struts 503 and 504 ensure that the UAV 102 rests in a stable,predictable, and repeatable manner when placed on the testing andcalibration system 500. The struts 503 and 504 are positioned such thateach calibration fiducial 214 on the UAV 102 is positioned within thefield of view of at least one of the cameras 502, when the UAV 102 isplaced on the testing and calibration system 500.

The cameras 502 are digital imaging sensors that can be used to capturedigital images of the calibration fiducials 214 of the UAV 102. FIG. 5Dillustrates a view of a wing 211 of the UAV 102 positioned over thesupport stand 501 with attached cameras 502 a and 502 b. FIG. 5E shows aclose up view of a camera 502, showing a lens opening 510 and lights511. Although the lights are shown attached to the camera body, inanother embodiment, the lights may be mounted separately from the camera502.

Calibration fiducials 214 are of two types. The movable fiducials 506are placed on the movable parts of the UAV 102, such as the controlsurfaces 213, payload bay doors, etc. The base fiducials 505 are placedclose to the movable fiducials 506, on a proximate part of the UAV 102that does not move. For example, in the illustrated embodiment the basefiducials 506 are placed on the wing 211, proximate to where the movablefiducials 506 are affixed to the control surfaces 213.

Each base fiducial 505 is paired with a corresponding movable fiducial506, and vice versa, and the cameras 502 are placed such that each pairof calibration fiducials is within the field of view of a camera 502.

The purpose of the fiducials is to present a distinct pattern that canbe easily identified in a digital image by a computer algorithm. In theillustrated example, the fiducials are distinct black and whitepatterns, which may be affixed to the UAV 102 as a decal or sticker (orany other visible paint, ink, pattern, texture, or the like). In theillustrated example, the pattern is an outer black square (the outerfiducial pattern) with a hollow interior containing additional symbols(the inner fiducial pattern). The black square (or any four-sidedpolygon or other appropriate shape) is used to determine the position ofthe fiducial, while the interior symbols help to uniquely identify thefiducial and give its orientation.

Alternative embodiments may also be used as fiducials. For example, inone embodiment the fiducials may be distinct color patterns on the UAV102, such as a sticker, paint, ink, texture, or decal. In anotherembodiment, the fiducials are a distinct pattern of small lights builtdirectly into the UAV 102 or affixed to the UAV 102 as part of aseparate assembly. In yet another approach a distinctive pattern ofsmall circles is used on each surface.

The testing and calibration system 500 may have additional componentsthat are not illustrated in the figures. Additional components mayinclude a testing and calibration computer for performing calculationsnecessary for the testing and calibration process, RFID sensors fordetecting RFID tags that are affixed to the UAV 102, lights forproviding uniform lighting of the fiducials, and a WIFI or ETHERNETmodule for communicating with other components of the UAS such as theUAV 102 and distribution management system 304. The testing andcalibration computer may be a separate computer server, a laptopcomputer, a PC, a handheld computer, an embedded computer that isintegrated into the testing and calibration system 500, or a softwaremodule that is part of the distribution center management system 304.

FIG. 5F illustrates one embodiment of a process used by the testing andcalibration system 500 to test and calibrate a movable part of the UAV102 (such as a control surface). All the movable parts of the UAV 102that require testing and calibration are prepared by affixing a fiducial(movable fiducial) to the part of the UAV that moves, and anotherfiducial (base fiducial) to an adjacent surface of the UAV that does notmove. These fiducials are typically applied to the UAV 102 as part ofthe manufacturing process. Once applied to a UAV 102, a fiducial willremain on the aircraft for a relatively long period of time, requiringreplacement only if the features of the fiducial are damaged orotherwise worn off.

The fiducials on the UAV 102 and the cameras of the testing andcalibration system 500 are used together to test and calibrate themovable parts of the UAV. Existing augmented reality systems have usedfiducials to aid in placing virtual objects in live video streams (see,e.g., Marker Tracking and HMD Calibration for a Video-based AugmentedReality Conferencing System, Hirokazu Kato and Mark Billinghurst,Faculty of Information Sciences, Hiroshima City University2HumanInterface Technology Laboratory, University of Washington, which ishereby incorporated by reference), however the testing and calibrationsystem 500 uses similar techniques to determine the orientation ofcontrol surfaces.

The first step of the process is to place the UAV 102 on the testing andcalibration system 500. This positions 530 at least one of the cameras502 such that the movable fiducial 506 and base fiducial 505 are in thecamera's field of view. The relative positions of the camera 502 and thebase fiducial 505 should be either acquired or computed by the testingand calibration system 500; or alternatively, the base fiducial andmovable fiducial's relative position should be computable by the testingand calibration system 500 from image data captured from the camera'sview and the camera's intrinsic parameters (such as focal length, lensdistortion, etc.).

In the illustrated embodiment, the support stand 501 may be used toensure that the fiducials will be in the field of view of the cameras502 and the relative positions are determined as a part of the laterprocess. However, in an alternative embodiment, the support stand 501can also be used to fix the cameras in a known position relative to thebase fiducials. In another alternative embodiment, the relativepositions of a camera and base fiducial can be determined with aninstrument that measures the distance between the camera view origin andthe base fiducial, and the angle of the camera view axis and thefiducial surface.

During testing and calibration, the real positions of the controlsurfaces of the UAV may be measured and compared to the positions ofthose control surfaces as reported by the actuator control system 204.In one embodiment, to compare these positions in a meaningful way, theactuator control system 204 of the UAV 102 may be sent a command (viaWiFi, Ethernet, or other means), from the testing and calibration system500, to move each control surface (or any other movable part that isbeing calibrated) to some set position, which can be used as the basisfor measurement in later steps. For example, in one embodiment, eachaileron of the UAV 102 may be commanded to move to a 45 degreedeflection angle relative to the wings or relative to a neutral or restposition.

After positioning 530, the camera 502 is activated to capture 535 animage that contains both the base fiducial and the movable fiducial. Inthe illustrated embodiment, the captured image is a black and whiteimage, which is sufficient since the fiducials for this embodiment are ablack and white pattern. If the fiducials contain color, the capturedimage may be a color image. The captured image data is then transferredto the testing and calibration computer for processing.

The testing and calibration computer processes the captured image todetermine 540 the fiducial features in the image. This can be done usinga number of different techniques. In the illustrated embodiment, thefiducial features are determined using a multi-step process. In thisembodiment, the captured image is black and white (e.g., the pixels ofthe image are in greyscale) and the fiducials are black and whitepatterns. The first step is to convert the captured image into binarydata using a thresholding algorithm, such as the algorithm described onpage 119 of Davies, E. Roy. Machine vision: theory, algorithms,practicalities. Elsevier, 2004, which is hereby incorporated byreference.

The binary data produced as output by the thresholding algorithm ispixel data where all the pixels are binary (e.g., pure black or purewhite, without any intermediate value pixels). These binary pixels areprocessed using a contour-finding algorithm, such as the algorithmdescribed in Fu Chang and Chun-jen Chen and Chi-jen Lu, A linear-timecomponent-labeling algorithm using contour tracing technique, ComputerVision and Image Understanding, 2004, which is hereby incorporated byreference. The contour-finding algorithm produces lines that formclosed, curved shapes. The largest, outer-most closed shape, correspondsto the outer fiducial pattern of the fiducials 505 and 506. Thisouter-most closed shape is the primary fiducial feature in the capturedimage, which may be used to determine the position of the fiducials inthe image.

After the fiducial features are determined 540 by the testing andcalibration computer, the 2D orientations of the fiducials in the cameraview are determined 545 based on the fiducial features. Although theword “orientation” is sometimes used to mean only the rotational pose ofan object, in the instant application, “orientation” may refer to boththe rotational pose and the location of the fiducial within the imagethat was captured by the camera. In this embodiment, to determine the 2Dorientations of the fiducials, the fiducial geometries in the image dataare determined based on the fiducial features.

Not all of the fiducial features are necessary to determine the fiducialgeometries as the shapes of the inner fiducial patterns are not relevantto the outer shape of the fiducials, and so may be ignored for thisportion of the process. The curved shape approximation of the outerfiducial pattern from the contour-finding algorithm is polygonized sothat the shape from the captured image approximately fits thestraight-sided polygonal shape (e.g., a 4-sided polygon, as in theillustrated example) of the outer fiducial pattern. An algorithm such asthe Ramer-Douglas-Peucker algorithm may be used to perform this step.See David Douglas & Thomas Peucker, Algorithms for the reduction of thenumber of points required to represent a digitized line or itscaricature, The Canadian Cartographer 10(2), 112-122 (1973), which ishereby incorporated by reference. At this point, though the rectangularfiducial has been identified within the image, it is not known if it isrotated relative to the view of the camera (e.g. the rectangular shapecould be rotated such that the top of the fiducial appears as the rightside of the rectangle detected in the captured image).

Once a polygonal shape that approximately fits the expected outerfiducial pattern is found by the testing and calibration computer, 2Dcoordinates of the corner vertices that represent the corners of thepolygonal shape are determined by intersecting the lines that form theapproximating polygonal shape. The pattern inside the fiducials isasymmetric and can be used to resolve any ambiguity related to the 2Dorientation of the fiducial within the image (e.g., the inner fiducialpattern may be used to determine if the fiducial is in an intended orexpected orientation, or whether it is rotated by 90 degrees, rotated by180 degrees, or rotated by 270 degrees, or rotated by any other amount).

Once the 2D orientation is determined 545, the 2D coordinates of thecorner vertices of the fiducials along with the known information aboutthe camera position is used to determine 550 the 3D orientation of thefiducials in real space. As mentioned before, the word “orientation” mayrefer to the rotational pose of an object and the 3D location of thefiducial in real space.

In the illustrated embodiment the 3D orientation in real space isrepresented by the 3D coordinates of the corner vertices in a cameracoordinate system, such as a coordinate system that has the camera atthe origin, a positive z-axis along the camera view axis, a positivex-axis to the right (based on the FIG. shown in 5D), and a positivey-axis down (based on the FIG. shown in 5D). Determining the 3Dcoordinates of the corner vertices of the fiducials in the cameracoordinate system can be accomplished by applying a modified form of thePerspective N-Point algorithm (PNP algorithm). A PNP algorithm may beused to determine the orientation of a camera based on the knownpositions of some fixed reference points in 3D real space and the 2Drepresentations of those fixed reference points in the image view of thecamera. However, in this case the PNP algorithm may be used to insteaddetermine the unknown 3D coordinates of the fiducial's corner verticesin the camera coordinate system based on the known intrinsic parametersof the camera and the 2D coordinates of the corner vertices that werepreviously determined. There are many types of PNP algorithms that canbe applied in this case. For example, a perspective 3-point algorithmmay be used.

Once the 3D orientation of the fiducials in real space is determined,the relative orientation between the base fiducial and movable fiducialcan then be determined 555. In the illustrated example, the orientationof each fiducial in real space is represented by the 3D coordinates ofthe corner vertices of that fiducial. The relative orientation of thosefiducials can be determined by calculating the difference in theorientations of the planes formed by those corner vertices, usingstandard 3D geometry techniques and matrix math.

Simplifications can also be introduced into the process of determiningthe difference between fiducial orientations, by taking into account thephysical constraints on movement of the movable parts resulting fromtheir design. For example, in the illustrated embodiment, the controlsurfaces move by rotating around a hinge that is fixed to the basesurface (e.g., the wing or tail of the UAV 102), and they are thereforeconstrained to move in a fixed curved arc. Thus, in the example,determining just the difference in angle between the fiducial surfacesmay be sufficient to also determine their relative positions in space.In one embodiment, the difference in angle between the fiducial surfacescan be represented in an angle-axis form to simplify other steps of theprocess. Since there are two possible axis-angle representations for anyrotation in real space, this ambiguity between the two representationscan be resolved by enforcing some arbitrary convention. For example, achoice of axis that gives a positive dot product with the normal to thebase fiducial may be enforced. Other axes or schemes may be chosen, solong as the scheme is used consistently for each fiducial.

The real position of the movable part may then be determined 560 basedon the difference in orientation between the base and movable fiducials.Since the base fiducial is placed on a surface that does not move, andthe movable fiducial is placed on a moving surface (such as a controlsurface), the real position of the moving surface can be determined 560,by the testing and calibration computer, based on the relativeorientation between the base fiducial and movable fiducial.

In the illustrated embodiment, the testing and calibration computerstores a “zero offset”, which corresponds to the relative position ofthe base fiducial and movable fiducial when the moving surface and basesurface are at a “zero position”. The zero position, is a starting pointfrom which the surfaces are measured. For example, in the illustratedexample, the control surface is an aileron coupled to a wing, and thezero position may be the position of the aileron when it is commanded tothe position of zero deflection relative to the wing. The zero offset isthe difference in orientation between the base fiducial and movablefiducial when the control surface is in the zero position.

The zero offset for each control surface on each UAV 102 may bedifferent, and the testing and calibration system 500 may store aseparate zero offset for each control surface of each UAV 102 in adatabase. When a UAV 102 is placed on the testing and calibration system500, the calibration system can detect which UAV is being tested, andbased on this, the testing and calibration system 500 can access theappropriate values of the zero offset for each control surface of thatUAV.

The testing and calibration computer utilizes a zero offset whencomputing the position of a control surface, because the angle betweenthe fiducials may not be zero at the zero position, and becauseimpreciseness of application of the fiducials to the measured surfaces,may be such that even when the control surface and base surface are at azero angle relative to each other, the fiducials applied to them may beat non-zero angles because of surface distortion. During calibration thereal position of the moving surface relative to the base surface isdetermined by taking the measured relative orientation between the basefiducial and the movable fiducial and offsetting it by the zero offset,to take into account possible initial misalignment between the base andmovable fiducials.

Hardware and software errors, mechanical slippages, and wear and tear onparts are detected by the testing and calibration computer by comparing565 the real position of the movable part to the position of the movablepart reported by the actuator control system 204 of the UAV 102 (the“reported position”).

The difference between the real position of the movable part and thereported position from the actuator control system 204 is called the“actuator variance”. If the actuator variance exceeds a threshold, thetesting and calibration computer may generate a notification 570.Otherwise, the testing and calibration computer will indicate that thetest has passed 575.

As described earlier, in one embodiment, the movable parts of the UAV102 are commanded to a position by the actuator control system 204, inpreparation for the testing and calibration process, in which case thereported position for a movable part will just be the positionpreviously commanded. In an alternative embodiment, the testing andcalibration system 500 can command the actuator control system 204 tomove a movable part of the UAV 102 through a sequence of discrete stepsand run the entire calibration process after each step. Running theprocess repeatedly like this can improve the calibration of a movablepart by testing the accuracy of its motion across its entire range ofmotion, instead of just at some fixed set point.

When the actuator variance exceeds the threshold this information cantrigger a notification to be sent or presented to a human operatorindicating that the part that is being tested requires repair orrecalibration. For example, when the actuator variance exceeds thethreshold, a signal can be sent to the operator interface 312 to informthe distribution center operator 107 that the tested control surfacemust be repaired or recalibrated.

To recalibrate a control surface, a human operator may make physicaladjustments to correct the discrepancy between the real position of thecontrol surface and the reported position. To check the accuracy of anadjustment, the human operator can use the testing and calibrationsystem to again test the control surface. If the calibration or repairwas sufficient, the actuator variance should fall under the acceptablethreshold.

The threshold that is used to decide whether or not to generate anotification can be selected based on how much actuator variance ismaterial to a particular vehicle and control surface. If a largeactuator variance is not an issue, the threshold can be set quite high(e.g. 5% variance, 10% variance, 15% variance, or any appropriatevariance) without there being much material effect on a UAV flight.However, for some control surfaces actuator variance must be kept verylow (e.g. 1% variance, 2% variance, 3% variance, or any appropriatevariance) in order to keep the vehicle safe. For such control surfaces,when a notification is generated, the UAV 102 must be taken out ofservice temporarily until the actuator that moves the control surfacecan be calibrated or repaired to fix the discrepancy between reportedposition and real position of the control surface.

The testing and calibration system can also use the actuator variance topredict future failures in mechanical equipment, including the actuatorsthat move control surfaces, by using a model that predicts equipmentfailure based on historical actuator variance measurements.

To use actuator variance to predict future failure in a particularcontrol surface, for example, a dataset related to control surfacefailure is first collected using the testing and calibration system orsimilar equipment.

To collect the dataset on control surface failure the actuator controlsystem is used to move the control surface for a testing period, and thetesting and calibration system is used to measure and store the actuatorvariance as it changes over the testing period, until the controlsurface suffers a mechanical failure. This is done many times with thesame type of control surface. The dataset will contain patterns in theactuator variance, and some of these patterns that occur just prior tomechanical failures will be predictive of those mechanical failures.

A predictive computer model can be may use the actuator variancepatterns to predict mechanical failure in the control surface. Thispredictive computer model can be manually developed by observing thepatterns in the actuator variance, or the predictive computer model canbe automatically developed or generated using machine learning andartificial intelligence techniques.

Either way, once the predictive computer model is developed, it can takeas input the actuator variance for a control surface measured over aperiod of time and can predict when the control surface will need to beserviced by a technician to prevent failure during a flight.

For example, a method to predict failures in a movable part of the UAV102 can be implemented as follows. First the testing and calibrationsystem can run a data collection process. The data collection processcan run over a period of time as the UAVs 102 are operated on missions.In this data collection process, the testing and calibration system isused after every flight to determine the actuator variance for aparticular movable part of interest. For example, the testing andcalibration system can be used on a particular control surface for whichfailure prediction is desired. For the data collection process, thetesting and calibration system may use the process described in FIG. 5Fon the control surface after each flight, except that instead ofcomparing the actuator variance determined by the process to a thresholdfor the purpose of generating a notification, the data collectionprocess will instead store each actuator variance measurement that isgenerated into a database along with performance data related to thecontrol surface from which the actuator variance measurement was taken.This performance data will include information about control surfaceerrors and failures.

Over time the testing and calibration system will store many actuatorvariances into the database along with corresponding information relatedto errors and failures. This database will serve as the training datafor a model development process. In the model development process thetraining data and a machine learning algorithm may be used to generate apredictive model. The predictive model can then be used to predictfailures and errors based on the historical actuator variances of acontrol surface, provided the control surface is of the same or similartype as the ones from which the data was collected in the datacollection process (or so long as the historical actuator variances areotherwise suitable for use with the control surface being tested).

Once the predictive model is developed we use the testing andcalibration system with the model to predict future failures and errorsin operating UAVs. To do this the testing and calibration system is usedto execute the data collection process for an operating UAV over aperiod of time. Again we store the actuator variances for a controlsurface of the UAV after every flight over the period of time. We feedthese historical actuator variances into the predictive model from themodel development process. When the predictive model determines that afailure or error is likely to occur in the control surface, based on thehistorical actuator variances, the testing and calibration system sendsa notification to the operator interface 312, which can inform thedistribution center operator 107 that it is time to service thatparticular control surface.

We claim:
 1. A system comprising: a camera configured such that a fieldof view of the camera is positioned to capture a base portion and amovable portion of an aircraft; a computer communicatively connected tothe camera, the computer configured to: determine a relative orientationbetween a base identification feature of the base portion and a movableidentification feature of the movable portion of the aircraft based onimage data received from the camera; determine a position of the movableportion of the aircraft based on the relative orientation between thebase identification feature and the movable identification feature, anda position of the base portion; compare the position of the movableportion to a reported position for the movable portion to determine avariance; and an aircraft control system communicatively connected tothe computer, the aircraft control system configured to monitor a stateof the aircraft based on the variance.
 2. The system of claim 1, whereinthe movable identification feature is coupled to a surface of themovable portion and the base identification feature is coupled to asurface of the base portion.
 3. The system of claim 1, whereinmonitoring the state of the aircraft based on the variance comprisesdetermining one or more of location, orientation, and speed of theaircraft based on a movement of the movable portion of the aircraft. 4.The system of claim 3, wherein the movement of the movable portion isdetermined based on a reported movement of the movable portion and thevariance.
 5. The system of claim 1, wherein determining the relativeorientation between the base identification feature and the movableidentification feature further comprises identifying, in the image data,fiducial features of the base identification feature and the movableidentification feature.
 6. The system of claim 5, wherein determiningthe relative orientation between the base identification feature and themovable identification feature further comprises determining fiducialgeometries of the base identification feature and the movableidentification feature based on the fiducial features.
 7. The system ofclaim 1, wherein the base identification feature and the movableidentification feature comprise an identifiable pattern.
 8. The systemof claim 1, wherein the base portion is adjacent to the movable portionand fixed relative to an aircraft reference frame.
 9. The system ofclaim 1, wherein the reported position is a value reported by anactuator control system of the aircraft.
 10. The system of claim 1,wherein the aircraft is an unmanned aerial vehicle, and the movableportion is a control surface of the unmanned aerial vehicle.
 11. Thesystem of claim 1, wherein the computer is further configured toanalyze, using a predictive model, a plurality of actuator variances todetermine the variance, wherein the predictive model is trained using ona plurality of historical actuator variances of the aircraft.
 12. Thesystem of claim 11, wherein the computer is further configured topredict a mechanical failure of the aircraft based on the variance andthe predictive model.
 13. The system of claim 11, wherein the computeris further configured to: execute a data collection process for a secondmovable portion of a second aircraft and store a second plurality ofactuator variances generated by the data collection process; applyingthe predictive model to the second plurality of actuator variances; andgenerating a notification based on output of the predictive model.
 14. Amethod comprising: capturing, with a camera, an image of a first portionof an aircraft including a first identification feature and a secondportion of the aircraft including a second identification feature, theimage comprising image data; determining, by a computer communicativelyconnected to the camera and using the image data, a position of thesecond portion of the aircraft based on a relative orientation betweenthe first identification feature and the second identification feature;and updating a state of the aircraft based at least on the position ofthe second portion of the aircraft.
 15. The method of claim 14, whereinthe first identification feature is coupled to a surface of the firstportion and the second identification feature is coupled to a surface ofthe second portion.
 16. The method of claim 14, further comprising:determining the relative orientation between the first identificationfeature and the second identification feature by: identifying, in theimage data, fiducial features of the first identification feature andthe second identification feature, and determining fiducial geometriesof the first identification feature and the second identificationfeature based on the fiducial features.
 17. The method of claim 14,wherein updating the state of the aircraft based at least on theposition of the second portion of the aircraft comprises determining oneor more of location, orientation, and speed of the aircraft based on theposition of the second portion of the aircraft.
 18. The method of claim14, wherein determining the position of the second portion of theaircraft comprises adjusting a reported position of the second portionof the aircraft using an actuator variance determined based on therelative orientation between the first portion and the second portion.19. The method of claim 18, wherein the reported position is a valuereported by an actuator control system of the aircraft.
 20. The methodof claim 14, wherein the first portion of the aircraft is adjacent tothe second portion and fixed relative to an aircraft reference frame.21. The method of claim 14, wherein the aircraft is an unmanned aerialvehicle, and the second portion is a control surface of the unmannedaerial vehicle.